Image based 3D Sensing and Modeling Technology for High-end Digital Cameras

of the Dissertation Image based 3D Sensing and Modeling Technology for High-end Digital Cameras by Xue Tu Doctor of Philosophy in Electrical Engineering Stony Brook University 2009 Image-based 3D sensing techniques have become increasingly important due to advances in digital camera technology. These techniques can be used in numerous applications like auto-focusing, 3D modeling, and photorealistic rendering. In this dissertation, we first present Depth from Defocus (DFD) technology used in camera auto-focusing. This technique models image formation process as a convolution operation and recovers the depth of an object in a scene by comparing two different iii blurred images. We extend the technique for cameras operating in macro-mode by adopting a new lens setting, a new magnification normalization algorithm, and a three-image DFD scheme in a certain error sensitive area. We then present a novel Shape from Shift-variant Blurring technique to relax the shift-invariance assumption in the DFD technique. This technique localizes the global shift-variant transform in a small neighborhood which elegantly models the actual imaging process. The process can be inverted to recover not only depth of the object, but also its slope and curvature. Both simulation and experimental results are presented to illustrate its high potential. We also present a novel multi-stereo camera system for 3D modeling. This multi-camera system consists of three recently designed stereo cameras. Each camera is equipped with a stereo adaptor which splits the incoming light and projects random patterns onto the object in a scene. The camera is programmed to capture two images consecutively in a short time. The first image is with the random dots projected and the second is the normal texture image. Through the Stereopsis technique, each camera is able to reconstruct a partial shape of the target object. We propose a new calibration algorithm specially tailored for its unique stereo camera architecture that significantly improves the accuracy. Then a fast Gaussian-Pyramid based stereo matching technique is presented for partial shape recovery. Three stereo cameras are placed about 1.2 m away from the object and at 45 degrees apart. After stereo matching, a fast registration algorithm is carried out to bring three partial shapes into a common camera coordinate system. Then we adopt a volumetric based shape integration method to extract an iso-surface from layered point clouds and construct a complete triangle mesh using Marching-Cube algorithm. Finally, we combine three texture images to render a photorealistic model on a computer iv